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Abstract 

Objective: To systematically review the literature on image-based telemedicine for medical expert consultation in acute 
care of injuries, considering system, user, and clinical aspects. 

Design: Systematic review of peer-reviewed journal articles. 

Data sources: Searches of five databases and in eligible articles, relevant reviews, and specialized peer-reviewed journals. 

Eligibility criteria: Studies were included that covered teleconsultation systems based on image capture and transfer with 
the objective of seeking medical expertise for the diagnostic and treatment of acute injury care and that presented the 
evaluation of one or several aspects of the system based on empirical data. Studies of systems not under routine practice or 
including real-time interactive video conferencing were excluded. 

Method: The procedures used in this review followed the PRISMA Statement. Predefined criteria were used for the 
assessment of the risk of bias. The DeLone and McLean Information System Success Model was used as a framework to 
synthesise the results according to system quality, user satisfaction, information quality and net benefits. All data extractions 
were done by at least two reviewers independently. 

Results: Out of 331 articles, 24 were found eligible. Diagnostic validity and management outcomes were often studied; 
fewer studies focused on system quality and user satisfaction. Most systems were evaluated at a feasibility stage or during 
small-scale pilot testing. Although the results of the evaluations were generally positive, biases in the methodology of 
evaluation were concerning selection, performance and exclusion. Gold standards and statistical tests were not always used 
when assessing diagnostic validity and patient management. 

Conclusions: Image-based telemedicine systems for injury emergency care tend to support valid diagnosis and influence 
patient management. The evidence relates to a few clinical fields, and has substantial methodological shortcomings. As in 
the case of telemedicine in general, user and system quality aspects are poorly documented, both of which affect scale up 
of such programs. 
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Introduction 

Rapid advances in telecommunication and information tech- 
nology have sparked the development of a variety of systems that 
allow for new forms and domains of medical consultation. During 
the past two decades, many broad reviews of telemedicine have 
been published, describing the state of knowledge and assessing - 
to some extent - the quality of the evidence at hand. Some reviews 
are wide ranging both in scope and geography [1,2], some are 



broad in scope but restricted to some countries [3], some deal with 
specific perspectives of application (like diagnostic and manage- 
ment decisions) [4,5], and rare ones look at costs [6]. Two recent 
systematic reviews added to the literature in this area: one assessed 
the effect of telemedicine on professional practice and on patient 
health care outcome [7] and the other was a systematic review of 
reviews about the effectiveness of telemedicine [8] . A consistent 
finding across reviews is that radiology, mental health, and 
dermatology are three domains of application with positive clinical 
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outcomes [4,5]. Yet, there are serious concerns that the 
evaluations conducted thus far are of rather poor methodological 
quality (e.g., design, methods, size/dimension) [9], with weak 
theoretical foundations, and limited to assessments of clinical 
management rather than patient recovery (and health). Also, little 
is known regarding their sustainability or the manner in which 
they can be implemented in other settings [4,5]. 

In the particular case of expert advice in the acute care of 
injured patients, expert consultation by telephone could be 
expected to significantly improve care access, quality and outcome 
by decentralising knowledge, speeding up and improving decision 
making and limiting patient transfer or expert displacement. This 
is encouraging as injury is an increasing cause of concern 
worldwide and it affects people from resource poor areas - where 
prognosis is not so good - to a far greater extent [10,1 1]. Reviews 
are available in this domain, but many are descriptive [12-14] or 
context specific [4,15]. A 2006 review focusing on accident and 
emergency telemedicine for primary care concluded that most 
studies conducted until then demonstrated technical feasibility and 
improved triage with an increasing range of local management, 
but few cost-effectiveness assessments were available [16]. Those 
reviews briefly introduced the role of telemedicine in the 
emergency department [14], current trends in the development 
and adoption of tele-medical adjuncts for injury control [17], 
potential applications/ functions of telemedicine for trauma and 
disaster management, and a review of systems from a US 
perspective [15]. Successful domains of application identified thus 
far are the transmission of computed tomography scans for urgent 
neurosurgical opinion and the transmission and interpretation of 
radiographs (usually peripheral limb films) for on going support of 
minor injury units [12,13]. In the case of burn injuries, studies are 
consistent on technical and clinical feasibility whereas less is known 
as regards clinical outcomes [18]. Systems have been evaluated in 
the main for their clinical accuracy, health care provider 
satisfaction, and follow-up of wound care [15]. 

Consulting those reviews helps us understand where the 
knowledge stands and what ethical and legal challenges are posed 
by the use of telemedicine in acute care. The knowledge at hand 
informs about various aspects of telemedicine, including those 
where experts are consulted and/or involved remotely in patient 
care. Yet, they provide limited assessments of the quality of the 
evidence thus far and they mix various types of telemedicine 
without specifying whether their conclusions actually apply to all 
of them. Although the field changes rapidly and new forms of 
teleconsultation enter the field of trauma care, those newer forms 
have not been reviewed in their own rights. 

Against this background, this systematic review was undertaken 
to 1. revisit and update the literature specifically on image-based 
telemedicine for medical expert consultation in acute care of 
injuries; and 2. systematically review the evidence at hand 
regarding system, user and clinical perspectives. Four main 
research questions are addressed: What is the system quality? 
What is the diagnostic validity? What is the effect on the 
management and clinical outcomes? What is the level of user 
satisfaction? 

Methods 

The procedures used in this review followed the PRISMA 
(Preferred Reporting Items for Systematic reviews and Meta- 
Analyses) Statement [19]. There is no published protocol for the 
systematic review, but the procedure is described in detail below. 

The DeLone and McLean Information System Success Model 
[20] was used as a framework to synthesise the results according to 



system quality, user satisfaction, information quality (diagnostic 
validity) and net benefits (management and clinical outcomes). 

Date sources and searches 

This systematic review includes studies that were published in 
articles from peer-reviewed journals. A systematic search identified 
potentially relevant articles in five electronic databases commonly 
used in this research area: MEDLINE, EMBASE, CINAHL, 
Cochrane Library and PsychlNFO. The databases were searched 
without a time limitation in June 2012, with the following terms (in 
tide and abstracts or as MeSH terms): "telemedicine", "mHealth", 
"m-health", "eHealth", "e-health", "mobile health", "emergen- 
cy", "emergencies"," injury", "injuries", "trauma", "acute burn", 
"acute burns". Relevant articles were also sought from the list of 
references of the reviews identified in the search, all articles from 
the archive of all online issues of the "Journal of Telemedicine and 
Telecare" and "Telemedicine and e-Health" Journal, starting 
from 2005 and from the list of references of the articles considered 
as eligible (see Figure 1 below). 

Study selection 

Articles on the subject of telemedicine for medical expert 
consultation in emergency care were included if they met the 
following criteria: evaluated the acute stage of injury/ trauma care, 
in emergency or pre-hospital settings; telemedicine intended to be 
used from point of care to specialist, and including image transfer; 
the system was assessed using human subjects; articles written in 
the English, French, Spanish, German or Nordic languages. 

Studies were excluded due to the following criteria: reviews, 
case studies or purely descriptive studies; done under extreme 
conditions (disaster situations, war zones, space, etc.); image 
transferred does not consist of trauma images, and if image 
transfer was done in conjunction with real-time interactive video- 
conferencing. 

First, two reviewers independendy evaluated the abstracts of all 
records identified in the initial databases search. If either reviewer 
could not exclude the article, the full-text was obtained and 
evaluated by the reviewers to assess eligibility. Secondly, additional 
references from reviews, journals and eligible articles were 
screened by title by one of the reviewers. If the title indicated 
relevance, the abstract and then the full-text were screened by two 
reviewers. 

Data extraction and quality assessment 

The articles were reviewed by all authors and after that a 
decision were made regarding the items that could be measured 
across most studies. Data was extracted on country, type of image, 
and the clinical focus that included the medical discipline and 
required expert, and the technology used for image treatment. 
From the four perspectives investigated, i.e. system quality, user 
satisfaction, diagnostic validity and clinical management, system 
quality and user satisfaction were seldom evaluated and the data 
gathered on those aspects related to the following. System quality 
considered above all on image quality, and time to complete 
different steps in the telemedicine process and user satisfaction 
included the perceived ease of use and usefulness of the system. In 
the case of diagnostic validity and management outcomes a wider range of 
data were compiled relative to the methodology of the studies, 
including e.g., sample size and statistics used, and to the results 
obtained. 

Attention was also paid to the methodological rigor of the 
studies by considering how various potential sources of bias were 
dealt with, based on "The Cochrane Collaboration's tool for 
assessing risk of bias" [21]: selection, performance, detection and 
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Figure 1. Flow chart of the literature search and screening process. 

doi:10.1371/journal.pone.0098539.g001 



attrition. The use of a "gold standard" was an additional criteria 
used for the studies dealing with diagnostic validity and 
management outcomes. 

Data extraction for each article involved at least two reviewers 
who independendy reviewed each article. After the individual 
assessments, the reviewers met two by two to discuss and agree on 
the data extraction and quality assessment of each article. 

Results 

Of the 331 articles identified in the database search, 16 were 
eligible for review, an additional 3 were obtained from screening of 
relevant references from reviews, and 5 were obtained from 
screening the references of the eligible articles (Figure 1). The 24 
articles describe 22 telemedicine systems, with two articles by 
Hsieh [22,23] describing one system, and two articles by Wallace 
[24,25] describing another. 

The articles were published between 1992 and 2011, and were 
mostly carried out in high-income countries. They appeared in 18 
different journals, two of which were from the telemedicine field 
and the others from medical journals (Table 1). 

Table 2 describes some general characteristics of the systems 
investigated, ordered according to their stage of development: 
feasibility studies; pilot or small-scale roll-out studies; and post- 
implementation studies. These characteristics include the condi- 
tions assessed, which belonged to different medical disciplines and 
mostly general traumas, followed by orthopaedic and hand 
injuries, that most often required the expertise of plastic surgeons, 
followed by radiologists and orthopaedic surgeons. Images were 



captured, transmitted and displayed through various technologies, 
and there were two types of images transmitted: radiological 
images and clinical photographs of the injury. As also shown in 
Table 2, most articles reported on management outcomes and 
diagnostic validity (perspective), while others assessed user 
satisfaction and system quality. 

Figure 2 represents the number of articles that report on 
different perspectives, within three different time periods. There is 
a trend whereby more recent articles seem to be focusing more on 
management outcomes and less on diagnostic validity. 

System quality 

12 articles assessed whether image transfer provides adequate 
support for injury acute care assisted by telemedicine. These 
articles evaluated the quality of the images [22,26-32] and how 
long it takes to complete different steps in the telemedicine process 
[22,23,32-35]. In some cases assessment of image quality was 
done using scales, and in others it was not clear how the 
assessments were made. Image quality was considered lower for 
telemedicine compared with original radiographs in a few of the 
studies [26-28,30]. Users in other studies expressed satisfaction 
with the telemedicine image quality [22,29-32]. In one of the 
studies, the quality of the telemedicine images was rated lower 
than the original radiographs, although the users were still satisfied 
with those images [30]. Operation time (from taking the image to 
reception of the image) was 3 to nearly 15 minutes [22,23,32] and 
the time for creating a file was 3 to 7 minutes [34]. One study 
indicated that telemedicine radiographs took longer time to read 
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than originals [33], and another one that telemedicine increased 
the time at the emergency department [35] . 

User satisfaction 

Five articles report user data [24,25,31,32,36]. They address the 
ease of use [24,25,3 1] and the perceived usefulness of the system 
for clinical decision making [24,25,32,36]. Three indicate that the 
data were gathered by questionnaire, [24,31,36] but none specifies 
how the question read, what the alternative answers were, and 
whether the questions - or answers - were standardized or 
validated. Overall, the studies report high levels of satisfaction and 
perceived ease of use. 

Diagnostic validity 

Table 3 presents the 1 7 articles that assessed diagnostic validity. 
Eight assessed systems at the feasibility stage, and nine at the pilot 
and post-implementation stages. The former all used radiological 
images, and had different designs whereby the assessors would 
either assess the images by one modality (i.e. either the original or 
digitized radiograph) [30,3 7], digitized images before the original 
ones [27-29], original images before the digitized ones [26], or 
mixed modalities where some assessors started with original and 
others started with digitized radiographs [33,38]. In one article 
[29] the description of the radiograph was also assessed and 
compared to the digitized and original radiographs. Assessments 
by both modalities were done one directly after the other [28,29], 
two weeks apart [26], at least four weeks apart [33,38], or 
six months apart [27]. All studies used a gold standard, and 
employed accuracy (sensitivity and specificity) measurements, 
Receiver Operating Characteristic (ROC), Kappa or McNamar's 
test. 

In five of the nine other studies that assessed systems at the pilot 
or post-implementation stages, the assessors considered cases 
through the telemedicine modality only [22,23,32,35,39]; in two, 
both telemedicine and on-site interpretations were done one 
after the other [40,41]. In two articles a telephone description 
was compared to the telemedicine modality [31,42]. 
Three evaluations used a gold standard [22,23,39], and statistical 
analysis included kappa, correlation coefficient and descriptive 
statistics, and accuracy analysis (sensitivity and specificity). Results 
showed generally good diagnostic accuracy, except in one study 
[39]. 

The main limitation of the evidence at hand was that 7 of the 
studies did not specify a gold standard [29,31,32,35,40-42] and 
that four studies did not use statistical tests to validate the diagnosis 
[31,32,35,40]. Convenience sampling were often used, in some 
studies clearly described [23,27,28,30,32,33,38], but in others not 
[22,26,29,37,40,41]. Even if this limits the general representative- 
ness of the studies, it may reflect specific or complicated diagnosis. 
Some of the articles did not clarify the performance of the studies 
[22,23,26,28,40,41] which made it difficult to review the rigor of 
those studies. 

Management and clinical outcomes 

Table 4 presents the 1 6 articles that assess the effect of image- 
based telemedicine on the clinical management of patients. In 
these articles, management plans after viewing digitized images 
were compared with written or oral descriptions 
[24,25,29,31,36,42-44], original radiographs or on-site examina- 
tion [29,40,41,43], and video [45]. Others estimated the 
consequence of misdiagnosis [39], or compared to the manage- 
ment suggested by the referring doctor [32] . 

None of the studies used a gold standard, and nine studies did 
not use statistical tests [22,31,32,34-36,39,40,43]. The selection of 
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Figure 2. Perspectives of articles by year of publication. 

doi:10.1371/journal.pone.0098539.g002 



cases was often not clearly described or explained 
[22,24,25,29,40,41], and four studies employed convenience 
sampling [32,34,36,43]. Furthermore, the performance of some 
studies was not clarified [22,36,40,41,43] (Table 4). 

Clinical outcome was assessed only in one of the recent studies 
[42]. In this article, mortality and Glasgow Outcome Score (GOS) 
at 6 months were compared between the patients who were 
transferred following telephone consultation and those transferred 
following telemedicine consultation (including images). Proportion 
of poor outcome (dead, vegetative or severely disabled) was higher 
in the group without telemedicine (32,1% vs 25,8%), but these 
differences were not significant. Overall mortality in both groups 
was the same (14,3%). 

Discussion 

Main findings 

This review dealt specifically with systems based on the transfer 
of images as a mean of consultation on acute injuries of various 
kinds. To date, by and large, those systems use above all radiologic 
images, they are evaluated at a feasibility stage or during small- 
scale pilot testing, and are put in place in a limited number of 
countries, all of which are high income. None of them are pre- 
hospital. 

Whereas the impact of the systems on diagnostic validity and 
management outcomes are often studied (see below). As is the case 
in other fields of telemedicine, the data at hand are less informative 
regarding both system quality and user satisfaction and we found 
only anecdotal economic evaluations [22-24,31] although meth- 
odological examples are available in the literature [46] . This may 
be due in part to the short life of some of the systems evaluated, 
but these knowledge gaps are now regarded as research areas to 
receive priority so as to allow policymakers and health care 
planners to make informed decisions (not least in low- and middle- 
income countries) [47-49]. Partly as a consequence, no standard 
methods of measurement emerge as regards systems quality or 
user satisfaction. From the reports available on user satisfaction for 
instance [24,25,31,32,36], ease of use and usefulness of telemed- 
icine are the two aspects studied and the studies are difficult to 
reproduce. 

As observed in previous reviews concerned with telemedicine 
for the support of medical care in general [1-9] or in some reviews 
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for emergency care of injuries in particular [15,16], the quality of 
the evaluations performed to date is somewhat poor, sometimes by 
using inadequate or no gold standard or an imprecise reference to 
validate the system, sometimes by their limited sample size and 
inappropriate statistical methods, and sometimes even by their 
poor reproducibility. 

Diagnostic validity 

Whether image-based telemedicine in acute care yield accurate 
clinical diagnosis was investigated for 16 systems and the majority 
relied solely on radiological images, some for injuries in general 
and others for specific body parts (e.g., hand or head). All but two 
feasibility studies [29,38] involved accuracy assessments. They 
were of varying size in terms of number of cases and a gold 
standard was used in all instances. Evaluations of systems at the 
small-scale phase [22,23,31,32,35,39-41] were more inclined to 
use a gold standard over time and in particular when they were of 
larger size (number of cases/images). The implemented system 
evaluated [42] did not use a gold standard. 

Not surprisingly, the general impression is that transmitted 
images, above all radiological ones and of a variety of body parts, 
can be accurately interpreted by specialists and that this has 
become more evident over time, i.e. while the technology itself 
allowed for better pictures, transmission and reading 
conditions. This finding can be interpreted as if consulting a 
radiologist or specialist is (has become) as accurate when using 
transmitted images as when using original ones. It is also of note 
that factors like age and experience of the teleexpert may impact 
on the level of accuracy just as do some characteristics of the 
injury. 

Management outcomes 

Whether telemedicine affected patient management was inves- 
tigated for 15 systems and all but one of them implied radiological 
images, some for injuries in general and others for specific body 
parts (e.g., hand or head). At the feasibility stage, three systems 
[29,36,45] out of 10 were assessed for their potential influence in 
that respect. All small scale system implementations assessed 
patient management (9 systems; [22,31,32,34,35,39-41,43]) and 
so did the three system evaluated once roll out (4 articles) 
[24,25,42,44]. 



□ System quality 
User satisfaction 

■ Diagnostic validity 

■ Management outcomes 
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Table 3. Diagnostic validity. 



Article 



Image type/ 
discipline 



Sample size 



Methodology 



Results 



Methodological 
limitations 



Preparatory/Feasibility studies 



Jacobs et al. 
2002 [26] 



Radiology/ 

General 

injury 



Krupinski 
et al. 
2000 [27] 



Radiology/ 

General 

trauma 



Images: 
n = 20 
10 

orthopantomographs 
(OPGs) (5 with and 5 
without fractures), 
10 occipitomentals 
(OMs) (5 with and 5 
without fractures) 
Assessors: n = 16 
8 Oral and 
maxillofacial 
surgeons (OMFS) 
and 8 accident and 
emergency (A&E) 
doctors 



Images: n = 40 films 
of bone trauma 
cases 

Assessors: n = 4 
2 orthopaedic 
surgeons and 2 
radiologists 



Procedure: 

Original radiographs viewed 
by OMFS and A&E doctors. 
Telemedicine images viewed 
by OMFS 2 weeks after 
assessing the originals. 
Outcome: 
Diagnostic accuracy 
Instrument: 
Questionnaire completed 
for each radiograph viewed. 
Confidence scores of 
diagnosis on a scale of 1-10. 
Statistics: 

Sensitivity and specificity in 

fracture diagnosis and 

identification of the position 

(overall and split into 10 

highest and 10 lowest quality 

radiographs) 

Gold standard: 

The assessment panel's 

assessment on the original 

radiographs. 



Procedure: 

Each assessor viewed the digital 
images. Six months later, they 
reviewed the original film. 
Outcome: 

Diagnostic accuracy and 

confidence in diagnosis. 

Instrument: 

A 6 point scale from 1(no 

lesion present, definite) to 6 

(lesion present, definite). 

Statistics: 

Receiver Operating Characteristic 
(ROC) analysis was performed 
on the confidence values. 
The Multi-Reader Multi-Case 
(MRMC) ROC analysis technique 
was used. 

Kappa: agreement between 
original film and digitized 
reading for each observer. 
Gold standard: 
Diagnoses by two radiologists 
who assessed the original films. 



Fracture diagnosis: 

Sensitivity: Original 
radiographs OMFS: 100/A&E: 
90; Telemedicine OMFS: 86 
Specificity: Original radiographs 
OMFS: 84/A&E: 77; 
Telemedicine OMFS: 80 
10 high quality 
radiographs - sensitivity and 
specificity of OMFS with 
telemedicine higher than 
A&E doctors with original 
radiographs. 

10 low quality radiographs, 
sensitivity and specificity of 
OMFS with telemedicine 
lower than A&E with original 
radiographs. 

Poor quality radiographs and 

frontozygomatic and 

infraorbital rim 

fractures were poorly 

diagnosed by telemedicine 

Position of the fracture 

more accurately 

assessed using original 

radiographs. 

Diagnosis by OMFS 

doctors using 

telemedicine was 

broadly comparable 

with fracture 

diagnosis by A&E 

doctors using 

original radiographs. 

Mean confidence 

scores: Original 

radiographs OMFS: 

6.8/A&E: 7.1; 

Telemedicine 

OMFS: 4.7 

No significant difference in 
diagnostic accuracy between 
original film and digital image. 
No significant differences in 
performance among the 4 
observers. 

Kappa values: 0.94 and 0.92 for 
the radiologists, 0.89 and 0.88 
for the orthopaedists. 
43% of the confidence ratings 
were exactly the same for film 
and photo viewing. 53% differed 
by only one category. Major 
differences were most often those 
images that were judged as poor 
quality and/or had poor framing. 
48 of 320 decisions (15%) were 
incorrect. 



- Selection: Not 
described 

- Performance: 
Original 
radiographs 
assessed before the 
telemedicine by the 
same assessor 



- Selection: 
Convenience 
sampling, although 
with clearly 
described inclusion 
criteria 

- Exclusion: Images 
where the original 
film was of poor 
quality were not 
evaluated 
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Image type/ 








Methodological 


Article discipline 


Sample size 


Methodology 


Results 


limitations 


Raikin et al. Radiology/ 


Images: n = 25 


Procedure: 


Overall, a significant improvement 


- Selection: Unclear 


1999 [29] Orthopaedic 


radiographs 


Assessors: 1) read a 


in the frequency of correct 


what the cases are 




Assessor: n = 4 


description of the 


diagnosis and treatment planning 


representative of 




4 orthopaedic 


radiographs, 2) viewed 


when digitized images were 


- Gold standard not 




surgeons 


the telemedicine image, 


used (91%) compared with 


described 






3) viewed original 


textual descriptions alone (48%) 








radiographs. 


(n<rn nni \ 








Outcome: 


Correct diagnosis and 








Diagnosis decision 


classification 








Instruments: 


- By the initial description of the 








Questionnaire after 


injury: 48%; by digitized 








each stage, regarding 


radiograph: 91%,; only 3 diagnosis 








change in diagnosis. 


changed after seeing the original 








Statistics: 


radiograph. 








Chi-square analysis 


- Significant difference between verba 








Gold standard: 


description and the other groups (p< 








Not specified 


0.001), but not between the digitized 





and original radiographs (p = 0.27). 
Significant differences between verbal 
descriptions and digitized radiographs 
in the surgeons' ability to appreciate: 

- Severity of injury: 34% vs. 97% (p< 
0.001) 

- Degree of comminution: 18% vs. 98% 
(p<0.001) 

- Degree of articular involvement: 23% 
vs. 93% (p<0.001) 

Where comminution and articular 
involvement could not be assessed, 
original films did not significantly add to 
the understanding. 



.arson et al. 


Radiology/ 


Images: n = 55 


Procedure: 


For subtle fractures, the sensitivity 


- Selection: 


1 998 [28] 


Spinal 


29 with normal 


Each assessor 


when using a teleradiology system 


Convenience 






findings 


viewed 


was similar to that of 


sampling, although 






and 26 with subtle 


the digitalized 


conventional radiographs. 


with clearly 






fractures 


image 


Sensitivity: original radiograph: 


described inclusion 






Assessors: n = 3 


and then the 


88%; digitized radiograph: 87% 


criteria 






radiologists 


original image. 


Specificity: original radiograph: 


- Performance: 








Outcomes: 


79%; digitized radiograph: 83% 


Same assessor 








Determination 


All cases were detected by 


evaluated the 








of 


at leas one radiologist; 6 of 26 


image by both 








presence of 


fractures were missed by at least 


modalities one right 








abnormalities 


one radiologist on the original and 


after the other 








Instruments: 


digitized images. 










The confidence 


ROC analysis showed that the 










scale 


differences between original and 










used the 


digitized images were not statistically 










following 


significant for any of the three 










values: 1 = 


radiologists. 










definitely normal, 












2 = probably 












normal, 3 = equivocal, 












4 = probably 












abnormal. 5 = definitely 












abnormal. 












Statistics: 












For comparison of sensitivity 












and 












specificity, a McNemar 












test for paired 












proportions was used. 












ROC analysis 












Gold standard: 












Consensus of two 












board-certified radiologists 
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Article 


Image type/ 
discipline 


Sample size 


Methodology 


Methodological 
Results limitations 


Reid et al. 


Radiology/ 


Images: n = 80 


Procedure: 


80% of the diagnosis of - Selection: 


1997 [37] 


Orthopaedic 


Cases with various 


Assessors reviewed 


the telemedicine and original Sampling not 






degrees of complexity. 


either teleradiology or original 


radiographs was concordant. A described 






Assessors: n = 4 


radiographs 


precise consensus diagnosis in 66% 






2 radiologists and 2 


Outcome: 


of the cases (78% for orthopaedists/55% 






orthopaedists. 


Diagnosis and relative certainty 


for radiologists). 








of the 


Precise diagnosis: 








diagnosis 


Orthopaedists: 93% of the original 








Instruments: 


radiograph readings and 80% of the 








Certainty measured on 


telemedicine readings (not significant 








a scale of 1-5 


difference). 








Accuracy of diagnosis; 


Radiologists: 70% of the original 








dichotomized in precise 


radiograph readings and 63% of the 








or wrong diagnosis. 


telemedicine readings (not sig 








Statistics: 


difference). 








McNemar's test. 


Statistically significant difference 








Gold standard: 


between orthopaedists and radiologists 








The diagnosis of the attending 


for reading original films, but not for 








orthopaedic surgeon who treated 


telemedicine films. 








the case. 


For those instances when the diagnosis 



was imprecise, the residents were aware 
of their Inability to make an accurate 
diagnosis. 

Significant relationship between 
diagnostic accuracy and certainty of 
diagnosis in orthopaedists reading 
radiographs via telemedicine. 
Confidence in diagnosis: 
Orthopaedists and radiologists had the 
same confidence in their diagnosis 
when reading original radiographs 
(p= 1.000), but differed significantly 
when reading via telemedicine 
{p = 0.039). 

Significant difference in certainty and 
accuracy between the two viewing 
modalities for both the orthopaedists 
and the radiologists. 

Intra-rater significant difference - Selection: 

between Convenience 

original films (superior) sampling, although 

and digitized images for with clearly 

3 of 4 radiologists. described inclusion 

Total fractures - statistically criteria, from two 

significant differences between sources 

original film and digitized images. 

Subtle fractures - ROC curves showed 

superior performance for original film 

with only three readers and only one 

was statistically significant. 

Non-subtle fractures -all readers 

performed better on original film, but 

the differences were statically 

significant for only two radiologists. 

For dislocations, calculations of 

sensitivity, specificity and accuracy were 

not significantly different for any reader 

between the original and digitized 

images. 



Wilson et al. 

1995 [33] 



Radiology/ 

General 

trauma 



Images: n = 180 
Radiographs of skeletal 
trauma patients. 
Assessors: n = 4 
4 radiologists 



Procedure: 

Each reader looked at 
one set of cases as 
original film and the 
other set of as 
digitized images. 
After at least 4 

weeks, cases seen as digitalized 
images 

were viewed as original and vice 

versa. 

Outcome: 

Identification of 

fractures and 

dislocations. 

Instruments: 

A 6 point scale (1 = definitely normal 
structure, 6 = definitely abnormal). 
Statistics: 

ROC analysis for each reader and 
each 

reading method. 
Accuracy, sensitivity 
and specificity of dislocations. 
Gold standard: 

Clinical and radiologic follow-up from 
medical records or consensus opinion 
of the original readings, study 
radiologists readings and final 
assessment by authors. 
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Image type/ 








Methodological 


Article 


discipline 


Sample size 


Methodology 


Results 


limitations 



Scott et a I. 
1993 [30] 



Radiology/ 
Orthopaedic 



Images: n = 1 20 

60 cases with fractures/ 

dislocations 

60 controls with similar 
age 

Assessors: n = 8 
7 senior radiology 
residents and 1 fellow 



Procedure: 

Each assessor viewed 60 random 
cases with the original film and then 
60 other cases using teleradiology 
(they did not view the same case in 
different modalities). 
Outcome: 

Diagnosis and confidence. 
Instruments: 

Confidence ratings of low, moderate, 
or high and a positive or negative 
reading, which corresponded to 6 
point scale from "almost definitely 
negative" to "almost definitely 
positive". 
Statistics: 
ROC analysis 

Frequency, accuracy, specificity and 

sensitivity 

Gold standard: 

Interpretations by three authors. 
When three or more readers 
misinterpreted a case, a consensus 
panel was used. 



with clearly 
described inclusion 
criteria 



Overall accuracy of the readers: - Selection: 

80.6% for original film interpretations Convenience 
and 59.6% for digitized readings (P< sampling, although 
.001}. 

Sensitivity: 78.5% for original film 
and 48.8% for digitized images (P< 
.001). 

Specificity: 83.2% for original film and 
72.3% for digitized images (P<.025). 
Original film readers produced 
significantly better results (p<0.05) 
than digitized readings for four of the 
eight readers in accuracy and for five of 
the eight in sensitivity. 
No significant difference in specificity 
for any of the individual readers. 
After the data were pooled, original film 
readings produced significantly better 
results for all three measures (accuracy, 
sensitivity and specificity). 
Accuracy and sensitivity were 
significantly less for digitized images 
within each of the 3 image quality 
categories, and especially low in 
moderate and high difficulty cases in 
the digital mode. 
ROC analysis showed a significant 
difference between original and 
digitized images. 



Yoshino et al. Radiology/Spinal 


Images: n = 50 


Procedure: 


2 of the 4 readers had statistically - Selection: 


1 992 [38] 


25 radiographs of 


Each assessor viewed the images 


significantly (p = 0.05) better fracture Convenience 




cervical spine fractures 


using both modalities, with at least 


detection using original radiograph. sampling 




and 25 radiographs 


four weeks apart. Two began with 


Pooled ROC scores for all readers were 




without fractures. 


original images and two began with 


0.904 for original radiographs and 0.868 




1 radiograph per 


telemedicine images. 


for telemedicine images. 




patient, selected by 


Outcome: 






the author. 


Diagnostic accuracy 






Assessors: n = 4 


Instruments: 






2 neuroradiologists, 1 


Level of certainty of fracture 






neuroradiology fellow, 


(1 = fracture definitely present to 






1 general radiologist. 


6 = fracture definitely not present). 








Location of fracture 








Statistics: 








ROC analysis from each reader and each 






reading method. 








Gold standard: 








Fractures were proven by autopsy, 








surgical findings, tomography or follow 








up examination. 





Pilot/small scale roll-out studies 



Diver et al. 


Clinical image and 


Images: n = 20 


Procedure: 


In 1 of 20 cases the face-to-face 


- Selection: Not 


2009 [40] 


radiology/Hands 


From trauma patients 


Each patient assessed at a trauma 


consultation highlighted patient 


described 






Assessors: n = 1 


clinic by a house officer (to mirror 


history details that were not 


- Performance: 






Plastic surgery registrar 


an AED doctor). The registrar viewed 


obtained through the consultation. 


Same assessor 








the image in combination with 


In 1 of 20 cases, a discrepancy in 


evaluated the 








telephone contact, and then the 


examination findings was identified 


image by both 








patients assessed the patients 


between the face-to-face 


modalities one right 








face-to-face. 


examination and the transmitted 


after the other 








Outcome: 


image. 


- No statistical tests 








Discrepancy in diagnosis 




- No use of a gold 








Instrument: 




standard 








Not mentioned 












Statistics: 












No 












Gold standard: 












The registrar assessed each patient in 












person. However, this was not used as 












gold standard in a statistical test. 
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Image type/ 








Methodological 


Article 


discipline 


Sample size 


Methodology 


Results 


limitations 


Chand- 


Radiology/General 


Images: n = 720 


Procedure: 


Both inter and intra- 


- Authors state the 


hanayingyoung 


injury 


From 93 patients 


Each assessor 


observer agreement were good 


limitation of having 


et al. 2007 [39] 




(59 emergency 


conducted two evaluations of the 


(kappa<0.60): 


more than one 






orthopaedic patients 


digital images, with 


Inter-rater agreement: kappa = 0.67 


source for the gold 






diagnosed with a 


two weeks in between. 


(good) 


standard 






non- or minimally 


Outcome: 


Intra-rater agreement: kappa = 0.68 








displaced fractures 


Determining the 


(good) 








and 34 age-matched 


presence of fracture 


Overall sensitivity was 78% at 1 st 








normal patients) 


and location of fracture. 


assessment and 80% at 2 n assessment 






Assessors: n = 4 


Instrument: 


Overall specificity was 57% at 1 st 








2 senior staff and 2 


Data collection form 


assessment and 54% at 2 n assessment 






junior 


Statistics: 


Overall accuracy was 66% at 1 st 








staff 


Kappa statistic was 


assessment and 65% at 2 n assessment. 








used to test for level of inter and 


Misdiagnosis: 










intra-observer agreements. 


- Overall misdiagnosis rate: 40%. 12% 










Sensitivity, specificity 


over-diagnosis, 27% under-diagnosis. 










and accuracy of each group of 


- No association was found between the 








assessors. 


experience of the assessors, the region 










Chi-squared to test the association 


of the fracture or the age group of the 








between variables and misdiagnosis. 


patients and the misdiagnosis rate. 










Gold standard: 












Clinical and 












radiographic follow up data. When not 










available, a panel of 3 specialists. 






Archbold et al. 


Radiology/ 


Images of 46 


Procedure: 


In 10 cases the MMS revealed 


- No statistical tests 


2005 [31] 


Orthopaedics 


consultations 


1) Assessment after telephone 


that the initial description of 


- No use of a gold 






Assessors: n = not 


referrals 


the injury was inaccurate with 


standard 






mentioned 


2) Assessment after 


respect to the actual injury. 








Trauma surgeons 


multi-media 










and referring 


consultations 










emergency physicians 


Outcome: 












Accuracy of injury description 












Instrument: 












Not mentioned 






Hsieh et al. 


Clinical image and 


Images: n = 128 


Procedure: 


Identified by all 3 


- Selection: 


2005 [23] 


radiology/Hands 


35 patients with 60 


The assessors reviewed 


surgeons: 


Convenience 






digit injuries 


image together with a 


- Amputation location in 


sampling, although 






Assessors: n = 3 


brief patient history. 


90% of the 60 digits. 


with consideration 






IJIa^Lli- ]> U 1 y 1 1 ]> 


Outcome: 


- Status of amputation 


of severity level 








- Injury extent 


level In 87% of the 60 


- Performance: 








- Ability to identify the 


digits. 


Authors participate 








location of amputation 


- Recognition of the 


as assessors 








- Status of amputation level 


presence of distal skin 










- Presence of distal ecchymosed 


ecchymosis along the 










skin along the digital arteries. 


digital artery: 79% sensitive 










Instruments: 


and 90% specific. 










A standard 


- Recognizing digital 










questionnaire. 


replantation potential was 










Statistics: 


90% sensitive and 83% specific. 





Sensitivity and specificity 
of remote diagnosis of 
distal skin ecchymosis 
and replantation 
potential - calculated 
when all 3 
surgeons agreed. 
Gold standard: 
On-site evaluation by 
the consultant 
attending plastic surgeon 
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Image type/ Methodological 
Article discipline Sample size Methodology Results limitations 



Hsieh et al. 


Clinical image and 


Images: n = 184 


Procedure: 


Remote diagnosis of the 


- Selection: Not 


2004 [22] 


radiology/Hands 


45 patients with 


The assessors reviewed 


skin defect: 79% sensitivity and 


described 






81 digital injuries 


image together with a 


71% specificity. 


- Performance: 






Assessors: n = 3 


brief patient history. 


Remote diagnosis of bone 


Authors participate 






junior plastic surgery 


Outcome: 


exposure: 76% sensitivity 


as assessors 






residents 


Identification of extent 


and 75% specificity. 










of injury (skin defect or 












bone exposure) 












Instruments: 












A standard wound 












questionnaire 












Statistics: 












Sensitivity and Specificity 












of remote diagnosis of wound 












descriptors (skin defect or bone 












exposure) were calculated 












under group agreement. 












Gold standard: 












Consultant surgeon viewed all 












patients in the emergency room shortly 










after the initial telemedicine referral. 






Poca et al. 


Radiology/Head 


Images: n = 90 


Procedure: 90 images were 


Of the 90 cases reviewed 


- No statistical tests 


2004 [35] 




teleradiological 


evaluated by the neuroradiologist 


by both assessors, the 


- No use of a gold 






examinations 


and neurosurgeon on call 


neuroradiologist detected 


standard 






Assessors: n = not 


independently. 


4 mild injuries that were 








mentioned 


Outcome: 


not detected by the neurosurgeon 








A neuroradiologist 


Discrepancy between 


on call. 








and 


the neuroradiologist 










the neurosurgeon 


and the neurosurgeon 










on call. 


on call. 












Instrument: 












Not mentioned 






Jones et al. 


Clinical image and 


Images: n = 82 


Procedure: 


Accuracy of transmitted image in 


- Selection: Not 


2004 [41] 


radiology/General 


Assessors: n = Not 


Cases were assessed by 


comparison to 


described 




trauma 


mentioned 


reviewing the telemedicine 


injury on examination 


- Performance: 






Trauma team: Senior 


images together with a 


was >97%. 


Assessments in the 






House Officer (SHO), 


conventional telephone referral. 


All surgeons had closely matched 


two modalities may 






registrar, consultant. 


and re-assessed on arrival to the 


scores for grade 


have been done by 








minor injury unit. 


of injury. 


the same team 








Outcomes: 


Overall, consultant 


- Exclusion: Some 








severity (grade of 


achieved the highest correlation 


images were not 








injury) 


coefficient 


evaluated- 








Instruments: 


when compared to the 


inadequate or lost 








A five-point scale, 


more junior members of 


data 








devised by the authors. 


the team. 


- No use of a gold 








Statistics: 




standard 








Correlation coefficient. 












Gold standard: 












Patient re-assessed on arrival. 












However, this was not used as gold 












standard in a statistical test. 






Pap et al. 


Clinical image and 


Images: n = 20 


Procedure: 


The clinical descriptions 


- Selection: 


2002 [32] 


radiology/Plastic 


Assessors: n = 4 


The assessors 


were clear and the 


Convenience 




surgery 


Attending plastic 


reviewed the digital images 


diagnoses precise in all instances. 


sampling, although 






surgeons 


together with 




in random order 








a telephone call. 




- No statistical tests 








Outcome: 




- No use of a gold 








Clinical description. 




standard 



Instruments: 

Not mentioned 

Post-implementation (Implemented system) 
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Image type/ 

Article discipline Sample size Methodology 

Goh et al. Radiology/Head Images: n = 31 

1 997 [42] 28 patients referred by 

telephone; 35 patients 

referred with 

teleradiology images. 

Assessors: n = not 

mentioned 

Neurosurgeons 



doi:1 0.1 371 /joumal.pone.0098539.t003 



The general impression is that consultation by telephone 
contributes to a change in management plan, including accuracy 
of triage /referral or a given treatment plan/procedure. This can 
be interpreted as if consulting a radiologist or specialist influences 
the management decisions made in acute care regarding injured 
patients. This in turn is conditional to transmitted images being as 
accurately interpreted as original ones (in case of radiology) or as 
at seeing the patient at bedside. Some of the data supporting this 
finding are perceptual (point of care or expert) and others are 
factual - a change was reported/observed. 

The data at hand however is of relatively poor quality with 
limitations in the use of gold standard, in study size (8 evaluations 
being based on less than 50 cases 2 having between 80 and about 
100 and the remaining 4 having over 150 cases), and in statistical 
methods. 

The review was limited in the way that most studies came from 
high-income countries and may not be representative of the 
conditions prevailing in low- and middle-income countries where 
this kind of research is much needed [47,50]. The research is 
based on well-established databases commonly used in similar type 
of reviews. We may have missed some studies captured in other 
types of databases but we assume that this loss is most likely to be 
small given the broad scope of the search itself. Furthermore, the 
review is restricted to articles written in the English, French, 
Spanish, German or Nordic languages. We also acknowledge the 
high likelihood of publication bias in favour of studies showing 
positive effect of telemedicine systems, which affects the state of 
knowledge [47]. Unfortunately, we were not able to describe and 
compare the technical features of the systems as much as we had 
expected in the beginning of the review process. The studies were 
published in different type of journals, but mainly medical ones, 
and the level of detail was very uneven. It goes without saying that 
it would be a great contribution to this field of research - and 
practice - if there were clear criteria to be met for the description 
of the systems evaluated. 

Way forward 

As availability of telecommunication and information technol- 
ogy expands, and penetration into low- and middle- income 



Methodological 
Results limitations 



There was generally good agreement - No use of a gold 

in CT diagnosis between the referring standard 

doctor and neurosurgical team. 

Only one case where the referring 

doctor missed a condition that had no 

impact on patient management in the 

acute phase. 



countries increases, image based telemedicine can play a key role 
in increasing access to expert advice in the acute care of injured 
patients. However, current evidence is generally of low method- 
ological quality and is limited in focus. In order to facilitate scale 
up of injury based acute care telemedicine systems - in a time of 
increasing burden of injury in many parts of the world - the 
literature is still incomplete. 

- Studies are needed to inform program development and 
implementation in general (to better understand barriers to large- 
scale implementation) and in resource poor settings in particular 
(where such systems are most urgently needed) [48,50,51]. 

- Research in this field needs to pay greater attention 
to user perspective (both healthcare professionals and 
patients) [48,51]. Failure to do so is a major threat to 
sustainability, as user acceptance is a prerequisite to implemen- 
tation. 

- Other aspects of telemedicine must be studied. Lack of 
basic evidence such as cost-effectiveness [47,50-53], effect on 
quality of care [48,52] and health outcome [47,50-53] has been 
highlighted as one of the major barriers to scaling up such 
programs. 

For scaling up telemedicine programs, several authors empha- 
size the importance of a common architectural design and 
interoperability of initiatives into existing health services [47,50- 
53]. National policies to ensure patient security and liability 
[51,53] and liaison of public and private partnerships [50-54] are 
other important elements for a broadening of initiatives. Policies 
could also ensure that strategies for monitoring and evaluation are 
included in the planning [52]. The creation of standard methods, 
instruments and measures would greatly assist interoperability and 
reproducibility of the myriad programs in use and being 
developed. 

Conclusions 

The present systematic review shows that image-based tele- 
medicine systems for injury emergency care tend to support valid 
diagnosis and influence patient management. However, the 
current evidence is generally of low methodological quality and 



Procedure: 

Neurosurgeons 
reviewed cases 
either by telephone 
consultation or 
by telephone 
consultation 
together with 
transmitted images. 
Outcome: 
Diagnostic 

accuracy - agreement 

between referring doctor and 

neurosurgeon. 

Instruments: 

Not mentioned 

Statistics: 

Fisher's exact 
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Table 4. Management outcomes assessment. 



Article 



Image type/discipline Sample size 



Methodology 



Results 



Methodological 
limitations 



Preparatory/Feasibility studies 

Mair et al. 201 1 [45] Radiology/General injury 



Images: n = 33 
Assessors: n = 20 
Emergency 
physicians 



Procedure: 

60 case reviews were 
conducted by video link or 
telephone call with viewing of 
digital images (PACS), in five 
sessions, held approximately 
four weeks apart. Some cases 
were presented by both 
modalities. 
Outcome: 

- A working management plan 

- Confidence in making the 
working management plan 

- Locally treated or transfer 
Instrument: 

Not mentioned 
Statistics: 

Kappa statistic was used to 
estimate within-observer 
agreement. 

Logistic regression (odds ratio) 



Proportion of patients - No use of a gold 

transferred was higher standard 

with PACS than video in 10 

cases, lower in 5 cases and 

the same in 6 cases. 

Proportion of patients 

transferred was higher 

when PACS was used for all 

except 5/20 doctors. 

The estimated odds for 

patient transfer were 56% 

lower when video was used 

instead of PACS (OR = 0.44 

95% CI 0.20-0.93). 

The estimated odds for 

patient transfer were 58% 

lower when a more 

experienced doctor was 

used instead of a less 

experienced one (OR = 0.42 

95% CI 0.17-1.02) 

Intra-agreement about 

transfer between 2 reviews 

by the same modality and 

doctor was 82%, which 

resulted in a kappa statistic 

of 0.54. 



Egol et al. 2003 [36] Radiology and 


Images from 1 1 


Procedure: 


The majority did not 


- Selection: 


clinical image/ 


orthopaedic 


Assessments were made after 


change their answers 


Convenience 


Orthopaedic 


emergency room 


clinical reading by the 


regarding the initial 


sampling 




consultations 


emergency room attending 


treatment with the 


- Performance: 




Assessors: n = 50 


physician and after digitized 


added information 


Authors mention the 




Voluntary 


images were shown. 


provided by telemedicine. 


difficulty of viewing 




physicians at a 


Outcome: 


- Admitting the 


in a large auditorium 




conference 


Initial patient management in 


patient: 


setting. 






terms of: 


83% remained 


- No statistical tests 






- admitting the patient 


unchanged. 


- No use of a gold 






- requiring surgery 


- Operative treatment: 


standard 






- coming to evaluate the 


78% remained 








patient 


unchanged. 








- needing more information 


- Need of more info 








Instrument: 


prior to making a 








Questionnaire before and after 


clinical decision: 70% 








viewing the images 


remained unchanged. 
Of 537 assessments, 
respondents agreed with 
the emergency room 
physician's interpretation 
in 264 instances (49%). 





Raikin et al. 1999 
[29] 



Radiology/Orthopaedic 



Images: n = 25 
radiographs 
Assessors: n = 4 
4 orthopaedic 
surgeons 



Procedure: 

Assessors: 1) read a description 

of the radiographs, 2) viewed 

the telemedicine image, 3) 

viewed original radiographs. 

Outcome: 

Treatment decision 

Instrument: 

Questionnaire 

Statistics: 

Chi-square analysis 



The difference in correct 
treatment plans between 
digitized images and 
actual radiographs was 
not significant (p = 0.27). 
It was possible to make 
a treatment plan, 
including need and type 
of surgery in 25% of the 
cases after verbal 
description. Treatment 
plan changed in 74% of 
the cases the decision to 
perform surgery and in 80% 
of the cases type of surgery 
planned would change, 
after seeing the digital 
image. An additional 5% 
would change after viewing 
the original radiograph. 



- Selection: Unclear 
what the cases are 
representative of 

- No use of a gold 
standard 
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Table 4. Cont. 



Methodological 

Article Image type/discipline Sample size Methodology Results limitations 



Pilot/small scale roll-out studies 



Abou Al Tout Clinical image/Hands 


Images: n = 460 


Procedure: 


In 19 cases, the 


- Selection: 


et al. 2010 [34] 


(and 4 videos}, from 


The emergency physicians 


management changed 


Convenience 




129 patients 


reviewed the patients and the 


due to the consultation. 


sampling 




Assessors: n = 8: 


teleexpert reviewed the 


4 times to modify medical 


- No statistical tests 




7 emergency 


transmitted images. 


prescription, 10 times to 


- No use of a gold 




physicians 


Outcome: 


modify an orthopaedic or 


standard 




1 hand surgeon 


Change in management/ 


surgical procedure, 5 








Observation 


times to modify referral 








Instrument: 


of the patient. 








Not mentioned 






Diver et al. Clinical image and 


Images: n = 20 


Procedure: 


In 1 of 20 cases there was 


- Selection: Not 


2009 [40] radiology/Hands 


From trauma 


Each patient assessed at a 


a difference between the 


described 




patients 


trauma clinic by a house 


management plan based 


- Performance: Same 




Assessors: n = 1 


officer (to mirror an AED 


on history/image analysis 


assessor evaluated 




Plastic surgery 


doctor}. The Image was 


and the plan following 


the image by both 




registrar 


transmitted to the registrar in 


face-to-face consultation. 


modalities one right 






combination with telephone 


5 of 20 patients could 


after the other 






contact 


have been adequately 


- No statistical tests 






Outcome: 


managed in a casualty 


- No use of a gold 






Differences between 


department, thus Image 


standard 






telemedicine and face-to-face 


analysis could have 








management decisions 


precluded the need for 








Instrument: 


transfer. 








Not mentioned 






Chandhanayingyoung Radiology/General injury 


Images: n = 720 


Procedure: 


Consequences of 


- No statistical tests 


et al. 2007 


From 93 patients 


Each assessor conducted two 


misdiagnosis: 


- No use of a gold 


[39] 


(59 emergency 


evaluations of the digital 


- Would have resulted in 


standard. 




orthopaedic patients 


images, with two weeks in 


mismanagement in up to 






diagnosed with a 


between. 


48% of the cases: Under 






non- or minimally 


Outcome: 


treatment in up to 45% of 






displaced fractures 


Estimated consequences of 


adult cases and 29% in 






and 34 age - matched 


misdiagnosis 


paediatric cases. 






normal patients} 


Instrument: 








Assessors: n = 4 


Data collection form 








2 senior staff and 2 










junior staff 








Archbold et al. 2005 [31] Radiology/Orthopaedics 


Images of 46 


Procedure: 


MMS consultation was 


- No statistical tests 




consultations 


1} Assessment after telephone 


felt to have changed the 


- No use of a gold 




Assessors: n = not 


referrals 


initial management of the 


standard 




mentioned 


2} Assessment after multi-media 


patients in 8/46 referrals. 






Trauma surgeons 


consultations 


Feeling the MMS 






and referring 


Outcome: 


consultations improved the 






emergency physicians 


Effect on patient management 


patient care: 34/46 cases 








Instrument: 


among trauma surgeons 








Questionnaire 


and 36/46 cases among 










emergency physicians. 




Hsieh et al. 2004 [22] Clinical image and 


Images: n = 184 


Procedure: 


1 5% of cases with 


- Selection: Not 


radiology/Hands 


45 patients with 81 


The consultant and residents 


disagreement of triaging 


described 




digital injuries 


reviewed image together with a 


between the 


- Performance: 




Assessors: n = 4 


brief patient history. 


teleconsultation and the 


Authors participate 




3 junior plastic 


Outcome: 


actual treatment by the 


as assessors 




surgery residents 


Triaging during remote 


attending surgeon. 


- No statistical tests 




1 consultant plastic 


consultation and actual 


25% of cases with 


- No use of a gold 




surgeon 


treatment according to on-site 


significant discordance 


standard 






inspection 


among residents; 








Instrument: 


difference partly 








Triage into 3 groups according 


attributable to the inability 








to severity and management 


to show instances of tiny 








plan: 


exposed digital bone or 








Group l-conservative treatment. 


tendon in some cases. 








Group ll-skin grafting or local 


1 5% with residents' 








flap coverage. 


agreement regarding the 








Group Ill-microsurgery such as 


triaging had a clinically 








replantation or free flap 


significant misinterpretation 






coverage. 


of an image. 
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Table 4. Cont. 



Methodological 



Article Image type/discipline 


Sample size 


Methodology 


Results 


limitations 


Jones et al. 2004 Clinical image and 


Images: n = 82 


Procedure: 


All surgeons had 


- Selection: Not 


[41] radiology/General trauma 


150 trauma referrals 


Cases were 


closely matched 


described 




Assessors: n = Not 


assessed by 


scores operative 


- Performance: 




mentioned 


reviewing 


priority. 


Assessments in the 




Trauma team: Senior 


the telemedicine 


The highest 


two modalities may 




House Officer (SHO), 


images together 


correlation 


have been done by 




registrar, consultant. 


with a conventional 


was seen in 


the same team 






telephone referral, 


scoring the 


- Exclusion: Some 






and re-assessed 


operative 


images were not 






on arrival to 


priority of 


evaluated: 






the minor injury unit. 


patient injuries 


inadequate or lost 






Outcome: 


(as compared 


data 






Operative priority. 


to injury 


- No use of a gold 






Instrument: 


severity) 


standard 






A five-point scale, 


Overall, consultant 








devised by 


achieved the 








the authors. 


highest correlation 








Statistics: 


coefficient 








Correlation coefficient. 


when compared 










to the more 










junior members 










of the team. 




Poca et al. 2004 [35] Radiology/Head 


Images: n = 160 


Procedure: The first 


Increase in 


- No statistical tests 




teleradiological 


90 images were 


tomographic 


- No use of a gold 




examinations 


evaluated by the 


examinations 


standard 




Assessors: n = not 


neuroradiologist and 


from 15% 






mentioned 


neurosurgeon on call 


in 1997, when 






A neuroradiologist 


independently. 


telemedicine 






and the 


Later, images were 


was not available 






neurosurgeon on 


assessed mostly by 


to 22% in 1998, 






call. 


the neurosurgeon. 


when telemedicine 








Outcome: 


was available. 








- Proportion of 


Decrease in the 








patients who 


number of patients 








received a 


transferred to a 








tomographic 


level 3 hospital 








examination. 


from 14% in 1997 








- Proportion of 


to 7% in 1998. 








referrals to a 


Increase in the 








level 3 hospital. 


number of patients 








- Mode of transferal 


treated at the referring 








(conventional 


hospital (27% in 








versus 


1997 and 34% in 








medicalized 


1998). Unnecessary 








ambulance) 


transfers were 








Instrument: 


avoided. 








Not mentioned 


Increase in 





number of patients 
referred with 
medicalized 
ambulances, when 
telemedicine was 
available. 



Pap et al. 2002 [32] 


Clinical image and 


Images: n = 20 


Procedure: 


The initial management 


- Selection: 




radiology/Plastic surgery 


From 20 patients 


The assessors 


suggested by the 


Convenience 






with 12 hand injuries 


reviewed the 


resident was modified 


sampling, although in 






Assessors: n = 4 


digital images 


on some occasions, 


random order 






Attending plastic 


together with a 


particularly with 


- No statistical tests 






surgeons 


telephone call. 
Outcome: 
Descriptive data on 
management decision. 
Instrument: 
Not mentioned 


complex problems. 


- No use of a gold 
standard 
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Table 4. Cont. 











Methodological 


Article Image type/discipline 


Sample size 


Methodology 


Results 


limitations 


Ricci et al. 2002 Radiology/General trauma 


Images of 108 


Procedure: 


26/123 (21%) plans 


- Selection: 


[43] 


patients with 1 23 


For each injury. 


were changed after 


Convenience 




acute fractures 


3 treatment 


viewing the radiograph 


sampling 




Assessors: n = not 


plans were 


images (12 acute 


- Performance: An 




mentioned 


formulated and 


management and 14 


author was the 




Attending 


recorded after: 


ultimate), but none were 


assessor 




orthopaedic surgeon 


1) traditional 


changed after viewing 


- No statistical tests 






verbal communication. 


the original radiograph. 


- No use of a gold 






2} digitized 


In 27/123 (22%) cases 


standard 






images were 


the attending physician 








reviewed. 


thought that review of 








3} review of 


images would be helpful 








the original 


to determine an accurate 








radiographs 


treatment plan: 








and physical 


In 15/27 (56%) cases plans 








examination. 


were changed (7 acute 








Outcome: 


management and 8 








Treatment plans 


ultimate). 








formulated 


In the 96 fractures were 








after each step. 


images were not thought to 






Two different 


be helpful, 11/96 (11%) 








types of 


plans were changed (5 








deviations 


acute management and 6 








from the 


ultimate) 








original plan 










were distinguished: 










1 . Changes in 










the acute 










management - any 










emergency 










department 










procedures. 










emergent operative 










procedures, or 










dispositions 










that were not part 










of the original plan. 










2. Changes in the 










ultimate management - changes 










to the original plan that did not 










affect emergency department 










treatment, emergent operative 










procedures, or the disposition of 










the patient. 










Instrument: 










Standardized data intake form. 






Post-implementation (Implemented system) 


Moya et al. Radiology/Head 


39 consultations 


Procedure: 


Before viewing the 


- No use of a gold 


2010 [44] 


(from 7 referring 


Assessment before and 


images, 25/39 (64%) 


standard 




hospitals) 


after viewing the 


would have been 






Assessors: n = not 


Web-based images. 


accepted for transport. 






mentioned 


Outcome: 


After viewing the images, 






Neurosurgeons 


Change in transfer and 


14/39 (36%) resulted in 








management decision. 


transfer. 








Instrument: 


44% (11/25) of the 








Three questions on the 


transports were avoided 








Web 


and the patients were 








site. 


managed locally. 








Statistics: 


The neurosurgeons 








Binominal distribution to 


recommended 








calculate the 95% CI. 


management changes in 








Fisher's exact test was 


44% (17/39) of all 








used to compare 


consultations. 








recommended management 










changes between those 










who were transported 










and not. 
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Methodological 

Article Image type/discipline Sample size Methodology Results limitations 



Wallace et al. 2008 [25] 


Clinical image and 


389 referrals where 


Procedure: 


Significant difference - Selection: The 




radiology/Plastic surgery 


telemedicine was 


Telemedicine assisted referrals 


(p = 0.004) in the selection of hospitals 






available (243 used 


compared to telephone only 


management of patients and units is not 






telemedicine) and 


referrals. 


with and without the described 






607 where only 


Outcome: 


availability of the - No use of a gold 






telephone referral 


Changes in patient 


telemedicine system. standard 






was available 


management 


Significantly fewer 






Assessors: n = not 


Instrument: 


patients needed further 






mentioned 


Not mentioned 


assessment or review and 






Receiving clinicians 


Statistics: 


more could be directly 








Chi squared test and CI 


booked for definitive care 








calculation 


on an operating list in the 










Day Surgery Unit, when 










telemedicine was available. 










Decrease in number of 










occasions when the referral 










hospital was unable to 










accept a referral due to a 










lack of capacity. 










No increase or decrease in 










patients being managed 










with only telephone advice. 










nor for patients admitted to 










their local hospital to await 










transfer to the referral 










hospital. 


Wallace et al. 2007 [24] 


Clinical image and 


389 referrals from 


Procedure: 


f~\ II C _l C 1 j_* TL 

Overall use of day - Selection: The 




radiology/Plastic surgery 


the telemedicine- 


Telemedicine assisted referrals 


surgery showed an 11% selection of hospitals 






equipped units. 


compared to telephone only 


increase in use. 28% for and units is not 






OAf\ ii carl 


referrals. 


telemedicine available described 






telemedicine) and 


Outcome: 


compared to 17%. - No use of a gold 






607 referrals by 


- Management of referrals 


Reduction in standard 






telephone 


(such as use of day surgery) 


unnecessary attendance 






Assessors: n = not 


- Accuracy of triage for those 


for hand trauma surgery 






mentioned 


attending day surgery for hand 


(13% in telephone referrals 






Receiving clinicians 


trauma 


vs. 3% in telemedicine 








- Management of patients with 


referrals). 








bum injuries 


The burns unit and day 








Instrument: 


surgery unit demonstrated a 








Not mentioned (used 9 


significantly improved 








management options) 


accuracy of triage. 








Statistics: 










Chi squared test and CI calculation 


Goh et al. 


Radiology/Head 


Images of 35 


Procedure: 


Therapeutic interventions - No use of a gold 


1997 [42] 




patients referred 


Neurosurgeons reviewed cases 


prior to the transfer standard 






with teleradiology 


either by telephone 


occurred in 3/28 






28 patients referred 


consultation or by telephone 


patients (10.7%) in the 






by telephone 


consultation together with 


group without 






Assessors: n = not 


transmitted 


teleradiology, and in 10/31 






mentioned 


images. 


patients (32.1%) in the 






Neurosurgeons 


Outcome: 


teleradiology group 








- Transfer time from the 


(p = 0.062). 








telephone decision to arrival at 


Incidence of secondary 








the neurosurgical unit. 


insults (adverse events) 








- Comparison of therapeutic 


occurred in 9/28 patients 








intervention - additional 


(32.1%) in the group 








measures advised by the 


without teleradiology, and 








neurosurgeon prior to transfer. 


in 2/31 patients (6.4%) in 








- Comparison of secondary 


the teleradiology group 








insults - adverse events that 


(p = 0.017). 








could affect the outcome. 










Instruments: 










Not mentioned 










Statistics: 










Fisher's exact 
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is limited in focus. User and system quality aspects are poorly 
documented, both of which affect scale up of such programs. 
Further work is required on quality, interoperability, and 
scalability. 

Supporting Information 

Checklist SI PRISMA Checklist. 
(DOC) 
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